Finite state model-based testing describes the behavior of a system (e.g., a software application) under test in terms of a finite state machine. Model-based testing generates software tests from explicit descriptions of an application's behavior. It is desirous to test software applications with a model-based utility because a model of an application is easily updated and is very adaptable to the changing conditions of a software application under test. As a software application evolves, static tests, such as a hard-coded program, have to be modified manually and recompiled whenever there is a change in the specification or functionality of the application. With model-based testing, the changes can be accounted for by simply changing the model. Furthermore, model-based testing enhances the quality of a software application by providing a means for a more accurate and thorough test.
The idea behind model-based testing is to establish a model for the behavior of the application under test. The behavior of the application is described through variables called “operational modes.” Operational modes dictate when differing inputs are applicable and how an application will react when the inputs are applied under different operational configurations of the application. For example, whether or not an application is currently running is a common operational mode. Typically, if the application is not running, the only action that a user can execute is to start the application. If the application is running, however, a user typically has a choice of multiple actions that could be executed.
A state is a valid combination of values of the operational modes. All of the possible states for an application under test are compiled into a model which is represented by a finite state transition table. Each entry in a state table consists of a current state, an input that is executed, and a next state that describes the condition of the application after the input is applied.
A simplified example of a software application from Microsoft Corporation of Redmond, Wash. is discussed to illustrate defining a simple finite state model to test the application. FIG. 1 shows two possible displays within a Clock window 100 for a “Windows” Clock application: an analog display 102 and a digital display 104. The Clock window also shows a user selectable option, “Settings” 106. Selecting “Settings” initiates a menu (not shown) that enables a user to select either a “Digital” or “Analog” option which changes an analog display to a digital display and vice-versa.
In the simplified example of the Clock application, there are four possible system inputs: (1) Start the Clock application, (2) Stop the Clock application; (3) Select Analog setting; and (4) Select Digital setting.
There are three rules for each of the four possible system inputs. The rules for each system input are as follows:
(1) Start
If the application is not running, the user can execute the Start command.
If the application is running, the user cannot execute the Start command.
After the Start command executes, the application is running.
(2) Stop
If the application is not running, the user cannot execute the Stop command.
If the application is running, the user can execute the Stop command.
After the Stop command executes, the application is not running.
(3) Analog
If the application is not running, the user cannot execute the Analog command.
If the application is running, the user can execute the Analog command.
After the Analog command executes, the clock is an analog display.
(4) Digital
If the application is not running, the user cannot execute the Digital command.
If the application is running, the user can execute the Digital command.
After the Digital command executes, the clock is a digital display.
The Clock application example will have two operational-modes: a system mode and a setting mode, each of which has associated values. The two operational modes and associated values are as follows:
System mode: {Running, Not—Running} to indicate whether the Clock application is running or not; and
Setting mode: {Analog, Digital} to indicate the format of the clock display.
From the above Clock application inputs, rules for the application inputs, and the operational modes, a finite state model is defined from all of the possible state combinations of the application under test. Table 1 below shows the state table for testing the Clock application. The names of the current and next states appear as a combination of operational mode values separated by a period. For example, the state “Running.Digital” means that the Clock application is running and that the display is digital.
TABLE 1Current StateApplication InputNext StateNot—Running.AnalogStartRunning.AnalogNot—Running.DigitalStartRunning.DigitalRunning.AnalogStopNot—Running.AnalogRunning.DigitalStopNot—Running.DigitalRunning.AnalogAnalogRunning.AnalogRunning.AnalogDigitalRunning.DigitalRunning.DigitalAnalogRunning.AnalogRunning.DigitalDigitalRunning.Digital@
Developing a model is an incremental process that requires simultaneously taking into consideration many aspects of an application under test. State model-based testing is cumbersome and tedious to implement for even a relatively simple software application when attempting to define a model for the application. As the state space increases for a large and complex application (i.e., the increasing number of possible states for an application under development), the model becomes increasingly burdensome to define. This is known as “state explosion”. It is unrealistic to create and maintain a model for an application by hand.
Continuing with the example of the Clock application, the model defined by the state table shown in Table 1 above can be graphically represented. FIG. 2 shows a state transition diagram 200 for the Clock model. The circles 202–208 are codes that represent the application states and the arrows are links that represent the application inputs 210–224. The direction of the arrows indicate an application input being applied to a current state of the application at the start of an arrow and the resultant next state at the end of the arrow. For example, a “Start” application input 210 applied to the current state “Not—Running.Analog” 202 results in a next state of “Running.Analog” 204.
The Clock application model defined by the state table shown above in Table 1 is analyzed by a graph traversal algorithm to develop a path, which is a test sequence, of the application inputs to be applied to the application under test in a particular order. Graph traversal algorithms are stand-alone programs that read in a state transition table from a file and follow a path through the model according to some traversal algorithm. Path segments for a model are graphically represented in the state transition diagram 200 illustrated in FIG. 2. Moving from one application state (i.e., nodes 202–208) to another following an application input (i.e., arrow links 210–224) is a path, or a segment of one. Graph traversal algorithms are well known and can be implemented to provide random sequenced paths, minimal duplication paths, or other optimized variations of path sequences. The result of analyzing the Clock application model with a graph traversal algorithm to find an optimal testing sequence is shown in Table below. The testing sequence traverses every application input in the state model as efficiently as possible. That is to say, the generated test sequence analyzes every application input (i.e., arrow links 210–224) with every associated application state (i.e., nodes 202–208) in the fewest number of link traversals as possible. For clarity, the Table 2 entries are labeled with the corresponding reference numbers from FIG. 2.
TABLE 2Test SequenceApplication InputNext Application State1Start(210)Running.Analog(204)2Analog(212)Running.Analog(204)3Digital(214)Running.Digital(206)4Digital(216)Running.Digital(206)5Stop(218)Not—Running.Digital(208)6Start(220)Running.Digital(206)7Analog(222)Running.Analog(204)8Stop(224)Not—Running.Analog(202)
The test sequence generated by a graph traversal algorithm, shown above in Table 2, is applied to a test driver that executes the test sequence on the software application under test. A test driver is a stand-alone program that reads a test sequence from a file (e.g., test—sequence.txt) and initiates testing code that can apply the test sequence to an application under test. The testing code runs the application as if a user was manipulating the application's user-selectable inputs. “Visual Test” has existing functions that can be called upon by a test driver to implement the testing code for an application test sequence. Table 3 below lists some examples of such functions that would be associated with developing the testing code for the Clock application example.
TABLE 3FunctionsDescription of FunctionRun(“C:\WINNT\System32\clock.exe”)Starts the Clock applicationWMenuSelect(“Settings\Analog”)Selects “Analog” on the“Settings” menuWSysMenu( 0 )Invokes “System” menu for theactive windowWFndWnd(“Clock”)Finds a window with “Clock”captionWMenuChecked(“Settings\Analog”)Returns “True” if “Analog” ischeck markedGetText(0)Returns the window title of theactive window
An example of programming code implemented by a test driver to test the first two actions in the test sequence for the Clock application example, “Start” and “Analog”, is shown below. The test execution program would read “Start” from the test sequence.txt file and execute the Visual Test function Run(“C:\WINNT\System32A\clock.exe”) associated with the “Start” action. The program would then read “Analog” from the file and execute the WMenuSelect(“Settings\Analog”) function associated with the “Analog” function.
open “test—sequence.txt” for input as #infile‘get the list of actionswhile not (EOF(infile))line input #infile, action‘read in an actionselect case actioncase “Start”‘start the Clockrun(“C:\WINNT\System32\clock.exe”)‘VT call to start Clockprogramcase “Analog”‘choose analog modeWMenuSelect(“Settings\Analog”)‘VT call to select menuitem Analogcase “Digital”‘choose digital modeWMenuSelect(“Settings\Digital”)‘VT call to select menuitem Digitalcase “Stop”‘stop the ClockWSysMenu (0)‘VT call to bring up systemmenuWMenuSelect (“Close”)‘VT call to select menuitem CloseEnd selectTest—oracle( )‘analyze Clock behaviorwend
After all of the actions in the test program have been executed, a test oracle function (Test—oracle( ) in the above programming code) can be called to determine if the application behaved as the model expected. A test oracle is a mechanism that verifies if the application has executed correctly. Another one of the benefits of model-based testing is the ability to create a test oracle from the state model. In the case of the simple Clock model example, a test oracle can verify whether the Clock is running or not, and whether the display is in an Analog or Digital mode.
The inventors have developed a system testing interface that provides a user with an easy to use interface to develop and generate a model for a software application under test, initiate the development of a test sequence with a graph traversal algorithm, and initiate the generation of a test execution program to test the application.